
United States Patent and Trademark Office 



UNITED STATES DEPARTMENT OF COMMERCE 
United States Patent and Trademark Office 

Address: COMMISSIONER FOR PATENTS 
P.O. Box 1450 

Alexandria, Virginia 223 1 3- 1 450 
www.uspto.gov 



APPLICATION NO. 


FILING DATE 


FIRST NAMED INVENTOR 


ATTORNEY DOCKET NO. 


CONFIRMATION NO. 


09/662,679 


09/15/2000 


Fernando C. M. Martins 


10559/195001/P8367 


1908 



20985 7590 06/03/2004 

FISH & RICHARDSON, PC 
12390 EL CAMINO REAL 
SAN DIEGO, CA 92130-2081 



EXAMINER 



BECKER, SHAWN M 



ART UNIT 



PAPER NUMBER 



2173 

DATE MAILED: 06/03/2004 



/> 



Please find below and/or attached an Office communication concerning this application or proceeding. 



PTO-90C (Rev. 10/03) 



Office Action Summary 



Application No. 

09/662,679 



Examiner 

Shawn M. Becker 



Applicant(s) 

MARTINS, FERNANDO C. M. 



Art Unit 

2173 



— The MAILING DATE of this communication appears on the cover sheet with the correspondence address — 
Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1.136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1)C3 Responsive to communication(s) filed on 19 March 2004 . 
2a)E3 This action is FINAL. 2b)D This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 11, 453 O.G. 213. 

Disposition of Claims 

4) E] Claim(s) 1-3.7-22.26 and 28-30 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) E3 Claim(s) 1-3. 7-22.26 and 28-30 is/are rejected. 

7) D Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) D The specification is objected to by the Examiner. 

10)D The drawing(s) filed on is/are: a)D accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 

Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 
1 1 )□ The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-1 52. 

Priority under 35 U.S.C. § 119 
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2.D Certified copies of the priority documents have been received in Application No. . 
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application from the International Bureau (PCT Rule 17.2(a)). 
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DETAILED ACTION 



This action is in response to communication filed 3/19/04. 



Claim Rejections - 35 USC §103 



1 . The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

2. Claims 1-3, 7-22, and 26 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
U.S. Patent No. 6,227,968 to Suzuki et al. (hereinafter Suzuki) and U.S. Patent No. 6,256,033 to 
Nguyen (hereinafter Nguyen). 

Referring to claims 1 and 26, Suzuki teaches a method and computer program product 
that receives audio data having a beat and forming beat data based on the audio data (col. 7, lines 
24-34). Suzuki teaches determining a gesture window within which a gesture (dance move) 
should occur, based on a specified time window relative to the beat data, and Suzuki plays the 
audio data and obtains gesture data during a time that the audio data is being played. See col. 7, 
lines 35-50. Also, see col. 1, lines 9-13, which describes how the player is to perform the action 
in time with the rhythm. 

Suzuki segments the gesture data according to the specified time window (time between 
beats), and Suzuki automatically determines information related to a gesture occurring within the 
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specified timing window (i.e. if the appropriate dance move/gesture occurred within the timing 
window for score keeping). See col. 7, lines 25-55 and col. 10, line 30 - col. 1 1, line 37. 

Suzuki does not teach that the gesture data is obtained through video data, wherein the 
video data is segmented to create a video clip of time including specified time window. 

However, Nguyen discloses a method and a computer program product for recognizing 
gestures contained in video data that segments video data to create a video clip based on timing 
data that indicates a specified timing window within which a gesture will occur. See col. 2, lines 
23-25, which states, "The frame captures the person in the action of performing the gesture at 
one moment in time" and col. 2, lines 28-34, which states, "These sequence of frame data sets 
taken over a period of time. . ." The method determines information related to a gesture 
occurring in the video clip only at the specified timing. See col. 3, lines 11-14, which describes 
"determining particular coordinates of the subject at a particular time ", (emphasis added). 

It would have been obvious to one of ordinary skill in the art to substitute the gesture 
recognition method (step-on base of a floor panel) in the dance game of Suzuki with the video 
gesture recognition method of Nguyen, such that the video data is segmented into video clips of 
specified timing according to beat data as supported by the beat timing window of Suzuki, 
because the step-on base of Suzuki only captures the movements of the player's feet, and does 
not recognize the movements of the rest of the body as intended in Suzuki (col. 1, lines 44-51), 
wherein the video gesture recognition method of Nguyen captures the movements of the entire 
body. 

Referring to claims 14, Suzuki teaches a system with an audio part that receives audio 
data having a beat and forming beat data based on the audio data (col. 7, lines 24-34). Suzuki 



Application/Control Number: 09/662,679 Page 4 

Art Unit: 2173 

teaches determining a gesture window within which a gesture (dance move) should occur, based 
on a specified time window relative to the beat data, and Suzuki plays the audio data and obtains 
gesture data during a time that the audio data is being played. See col. 7, lines 35-50. Also, see 
col. 1, lines 9-13, which describes how the player is to perform the action in time with the 
rhythm. 

Suzuki segments the gesture data according to the specified time window (time between 
beats), and Suzuki automatically determines information related to a gesture occurring within the 
specified timing window (i.e. if the appropriate dance move/gesture occurred within the timing 
window for score keeping). See col. 7, lines 25-55 and col. 10, line 30 - col. 11, line 37. 

Suzuki does not teach that the gesture data is obtained through video data, wherein the 
video data is segmented to create a video clip of time including specified time window. 

However, Nguyen discloses a system with a processor (Fig. 1, 102) for recognizing 
gestures contained in video data that has a temporal segmentor that segments video data to create 
a video clip based on timing data that indicates a specified timing window within which a gesture 
will occur. See col. 2, lines 23-25, which states, "The frame captures the person in the action of 
performing the gesture at one moment in time" and col. 2, lines 28-34, which states, "These 
sequence of frame data sets taken over a period of time. . ." The method determines information 
related to a gesture occurring in the video clip only at the specified timing. See col. 3, lines 1 1- 
14, which describes "determining particular coordinates of the subject at a particular time ", 
(emphasis added). Nguyen also discloses a recognition engine, in communication with the 
temporal segmentor, to determine if the video clip contains a predefined gesture, only within a 
specified timing window. See col. 1, lines 17-23 and col. 3, lines 1 1-14. 
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It would have been obvious to one of ordinary skill in the art to substitute the gesture 
recognition system (step-on base of a floor panel) in the dance game of Suzuki with the video 
gesture recognition system of Nguyen, such that the video data is segmented into video clips of 
specified timing according to beat data as supported by the beat timing window of Suzuki, 
because the step-on base of Suzuki only captures the movements of the player's feet, and does 
not recognize the movements of the rest of the body as intended in Suzuki (col. 1, lines 44-5 1), 
wherein the video gesture recognition system of Nguyen captures the movements of the entire 
body. 

Referring to claims 2-3, the combination of Suzuki and Nguyen, supra, describes how 
Hidden Markov Models are used to determine a probability that each of a plurality of predefined 
gestures are performed within a timing window. See col. 5, lines 13-45 of Nguyen. 

Referring to claim 7, 17-18, and 21, the dance game of Suzuki teaches displaying a target 
gesture to be performed by the subject. The target gesture is a dance move. See col. 12, lines 
57-61. The target gesture is displayed on the display subsystem (monitor; col. 12, line 59). 

Referring to claims 8-10, the combination of Suzuki and Nguyen, supra, teaches each 
video clip contains video frames (Nguyen at col. 7, line 22), and in each frame, the moving 
regions are identified (Nguyen at col. 8, lines 58-61). Feature vectors (array of key points) are 
generated for each video frame of the video clip. See Nguyen at col. 8, line 49 - col. 9, line 46, 
which describes how significant positional coordinates are extracted from each frame to make a 
comparison to the known gesture coordinates. 
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Referring to claims 11-12 and 19-20, the method of Suzuki and Nguyen, supra, generates 
and displays a score based on whether a target movement (gesture) was performed. See Suzuki 
at col. 1 1, lines 37-41. The object of the Suzuki dance game is to achieve the highest score by 
performing the appropriate gestures. 

Referring to claims 13 and 22, the combination of Suzuki and Nguyen, supra, discloses 
that the recognition engine is configured to recognize predefined gestures and that determining if 
the video clip contains a target gesture includes generating a gesture probability vector (array) 
having a plurality of elements, each element being associated with one of a predefined gestures 
and representing a probability that the video clip contains each of the associated predefined 
gestures. See col. 1 1, lines 30-49 and col. 10, lines 18-37. 

Referring to claim 15, the recognition engine of Suzuki and Nguyen, supra, includes a 
plurality of Hidden Markov Models. See Nguyen at col. 5, lines 13-18. 

Referring to claim 16, the system of Suzuki and Nguyen, supra, includes a video source 
(Nguyen at Fig. 2, 200), in communication with the temporal segmentor, to provide the video 
data to the temporal segmentor. 

3. Claims 28-30 are rejected under 35 U.S.C. 103(a) as being unpatentable over Nguyen and 
Suzuki. 

Referring to claim 28, Nguyen teaches an A/V processing system that includes a video 
source (camera; Fig. 2, 200). Nguyen discloses a computer program product for recognizing 
gestures contained in video data, comprising instruction operable to cause a programmable 
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processor, in communication with the video source to segment the video clip and automatically 
determine if the video clip contains a predefined gesture. See col. 1, lines 17-23. 

Nguyen determines a gesture window within which a gesture should occur. See col. 2, 
lines 23-25, which states, "The frame captures the person in the action of performing the gesture 
at one moment in time" and col. 2, lines 28-34, which states, "These sequence of frame data sets 
taken over a period of time. . ." The method determines information related to a gesture 
occurring in the video clip only at the specified timing. See col. 3, lines 11-14, which describes 
"determining particular coordinates of the subject at a particular time ", (emphasis added). 

Nguyen does not explicitly teach an audio source to provide audio data having a beat 
from which beat data is extracted, such that the a specified time window of the segments are 
based on the beat data or obtaining the video data during a time that the audio signal is being 
produced. However, Suzuki teaches a dance game, which is intended to make the player use his 
entire body to create rhythm sensations (col. 1, lines 47-50). The dance game of Suzuki teaches 
that audio data is received and the beat data is extracted to create a specified timing window (col. 
7, lines 24-34). Suzuki determines if a predefined gesture (dance move) is performed by 
detecting dance steps on a step-on base only within a specified timing window related to the beat 
data. For example, see col. 14, lines 6-18. It would have been obvious to one of ordinary skill in 
the art at the time of the invention to modify the gesture recognition method of Nguyen to be 
used in the dance game of Suzuki, such that the video data is obtained during a time that the 
audio data is played and the specified timing window for seeing if a gesture occurred is related to 
beat data extracted from audio data as supported by Suzuki. One would have been so motivated 
in order to compare all movements of a player (i.e. arms) to an appropriate dance move because 
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the step-on base of Suzuki only captures the movements of the player's feet, and does not 
recognize the movements of the rest of the body as intended in Suzuki. 

Referring to claim 29, Nguyen teaches the computer program product includes instruction 
operable to cause the programmable processor to perform a Hidden Markov Model process to 
determine if the video clip contains the predefined gesture. See col. 5, lines 13-18. 

Referring to claim 30, Nguyen discloses a display (monitor; Fig. 2, 208) to display 
information based on whether the video clip contains the predefined gesture. See col. 6, lines 
36-44, which describes how figures on the display can be augmented if the gesture is recognized 
as a predefined gesture. 

Response to Arguments 

4. Applicant's arguments filed 3/19/04 have been fully considered but they are not 
persuasive. 

5. In response to applicants argument that the examiner's conclusion of obviousness is 
based upon improper hindsight reasoning, it must be recognized that any judgment on 
obviousness is in a sense necessarily a reconstruction based upon hindsight reasoning. But so 
long as it takes into account only knowledge which was within the level of ordinary skill at the 
time the claimed invention was made, and does not include knowledge gleaned only from the 
applicant's disclosure, such a reconstruction is proper. See In re McLaughlin, 443 F.2d 1392, 
170 USPQ 209 (CCPA 1971). 

Applicant argues that the object of Suzuki (i.e. providing a dance game that creates 
rhythm sensations by using the entire body) had been accomplished by the step-on base of 
Suzuki; therefore, there is no suggestion to use another type of user input. While the object of 
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Suzuki may have been accomplished with the step-on base, the step-on base is but one 
embodiment for recognizing user input. Applicant argues that, at the time of the invention, user 
input relied upon mechanical means and not video capture. However, Suzuki suggests full body 
gestures as a form of user input (col. 1, lines 44-51), and Nguyen makes it clear that user input 
could be achieved by video capture and gesture recognition at the time of the invention. 
Therefore, one of ordinary skill in the art, with the references of Suzuki and Nguyen before him 
at the time of the invention would have been motivated to substitute the mechanical input of the 
step-on base of Suzuki with the gesture recognizing video capture of Nguyen in order to capture 
full body movements, such as the arm movements (i.e. flapping, col. 6, lines 13-36 of Nguyen). 
Nguyen further states, "performing a recognized gesture is essentially the same as pressing a key 
on the keyboard or clicking a button on a mouse" (col. 6, lines 34-36) which are the type of 
mechanical inputs that Applicant states were not known to have video gesture recognition as 
alternative inputs at the time of the invention. 

Applicant argues that the Suzuki does not form beat data based on received audio data 
having a beat. It should be noted that claim 1 does not recite that the formed beat data is based 
on the beat of the audio data, but simply that the audio data has a beat, and that beat data is 
formed based on the audio data (not the beat data). Furthermore, Suzuki teaches forming beat 
data for each type of data (col. 7, lines 29-34); therefore the beat data is formed based on the 
audio data (type of audio). 

Applicant argues that Nguyen does not teach segmenting captured data into a clip, 
because Nguyen examines individual frames. However, Nguyen examines consecutive 
individual frames over a period of time (col. 2, lines 27-28), therefore the consecutive frames 



Application/Control Number: 09/662,679 Page 10 

Art Unit: 2173 

make up a video clip. Applicant's argument that the segmenting of Nguyen is manually dictated 
is taken out of context with the gesture recognition. The cited section (col. 11, lines 23-26) 
describes training the gesture recognition system (i.e. inputting new gestures), not recognizing 
the already known gestures. 

Conclusion 

6. THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1 .136(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the mailing 
date of this final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Shawn M. Becker whose telephone number is 703-305-7756. 
The examiner can normally be reached on M-Th 8:00 - 5:30 and alternating Fridays. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, John W. Cabeca can be reached on 703-308-3 116. The fax phone number for the 
organization where this application or proceeding is assigned is 703-872-9306. 
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Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). /J 



smb 




JOHN CABECA 
SUPERVISORY PATENT EXAMINER 
TECHNOLOGY CENTER 2100 



